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Abstract 

We describe an approach for unsupervised learning of a generic, distributed sen¬ 
tence encoder. Using the continuity of text from books, we train an encoder- 
decoder model that tries to reconstruct the surrounding sentences of an encoded 
passage. Sentences that share semantic and syntactic properties are thus mapped 
to similar vector representations. We next introduce a simple vocabulary expan¬ 
sion method to encode words that were not seen as part of training, allowing us 
to expand our vocabulary to a million words. After training our model, we ex¬ 
tract and evaluate our vectors with linear models on 8 tasks; semantic relatedness, 
paraphrase detection, image-sentence ranking, question-type classification and 4 
benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf 
encoder that can produce highly generic sentence representations that are robust 
and perform well in practice. We will make our encoder publicly available. 


1 Introduction 

Developing learning algorithms for distributed compositional semantics of words has been a long¬ 
standing open problem at the intersection of language understanding and machine learning. In recent 
years, several approaches have been developed for learning composition operators that map word 
vectors to sentence vectors including recursive networks HI, recurrent networks ||2], convolutional 
networks 001 and recursive-convolutional methods sia among others. All of these methods 
produce sentence representations that are passed to a supervised task and depend on a class label in 
order to backpropagate through the composition weights. Consequently, these methods learn high- 
quality sentence representations but are tuned only for their respective task. The paragraph vector 
of IT) is an alternative to the above models in that it can learn unsupervised sentence representations 
by introducing a distributed sentence indicator as part of a neural language model. The downside is 
at test time, inference needs to be performed to compute a new vector. 

In this paper we abstract away from the composition methods themselves and consider an alterna¬ 
tive loss function that can be applied with any composition operator. We consider the following 
question: is there a task and a corresponding loss that will allow us to learn highly generic sentence 
representations? We give evidence for this by proposing a model for learning high-quality sentence 
vectors without a particular supervised task in mind. Using word vector learning as inspiration, we 
propose an objective function that abstracts the skip-gram model of fS] to the sentence level. That 
is, instead of using a word to predict its surrounding context, we instead encode a sentence to predict 
the sentences around it. Thus, any composition operator can be substituted as a sentence encoder 
and only the objective function becomes modified. Figure [^illustrates the model. We call our model 
skip-thoughts and vectors induced by our model are called skip-thought vectors. 

Our model depends on having a training corpus of contiguous text. We chose to use a large collection 
of novels, namely the BookCorpus dataset El for training our models. These are free books written 
by yet unpublished authors. The dataset has books in 16 different genres, e.g., Romance (2,865 
books). Fantasy (1,479), Science fiction (786), Teen (430), etc. Table [T] highlights the summary 
statistics of the book corpus. Along with narratives, books contain dialogue, emotion and a wide 
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got back home <eos> 



<eos> This was strange 


Figure 1: The skip-thoughts model. Given a tuple (si_i, Si, Si+i) of contiguous sentences, with Si 
the i-th sentence of a book, the sentence Si is encoded and tries to reconstruct the previous sentence 
Si_i and next sentence Si+i. In this example, the input is the sentence triplet / got back home. I 
could see the cat on the steps. This was strange. Unattached arrows are connected to the encoder 
output. Colors indicate which components share parameters, (eos) is the end of sentence token. 


# of books 

# of sentences 

# of words 

# of unique words 

mean # of words per sentence 

11,038 

74,004,228 

984,846,357 

1,316,420 

13 


Table 1; Summary statistics of the BookCorpus dataset Q. We use this corpus to training our 
model. 

range of interaction between characters. Furthermore, with a large enough collection the training 
set is not biased towards any particular domain or application. Table shows nearest neighbours 
of sentences from a model trained on the BookCorpus dataset. These results show that skip-thought 
vectors learn to accurately capture semantics and syntax of the sentences they encode. 

We evaluate our vectors in a newly proposed setting: after learning skip-thoughts, freeze the model 
and use the encoder as a generic feature extractor for arbitrary tasks. In our experiments we con¬ 
sider 8 tasks: semantic-relatedness, paraphrase detection, image-sentence ranking and 5 standard 
classification benchmarks. In these experiments, we extract skip-thought vectors and train linear 
models to evaluate the representations directly, without any additional fine-tuning. As it turns out, 
skip-thoughts yield generic representations that perform robustly across all tasks considered. 

One difficulty that arises with such an experimental setup is being able to construct a large enough 
word vocabulary to encode arbitrary sentences. For example, a sentence from a Wikipedia article 
might contain nouns that are highly unlikely to appear in our book vocabulary. We solve this problem 
by learning a mapping that transfers word representations from one model to another. Using pre¬ 
trained word2vec representations learned with a continuous bag-of-words model |(8l, we learn a 
linear mapping from a word in word2vec space to a word in the encoder’s vocabulary space. The 
mapping is learned using all words that are shared between vocabularies. After training, any word 
that appears in word2vec can then get a vector in the encoder word embedding space. 


2 Approach 

2.1 Inducing skip-thought vectors 

We treat skip-thoughts in the framework of encoder-decoder models Q That is, an encoder maps 
words to a sentence vector and a decoder is used to generate the surrounding sentences. Encoder- 
decoder models have gained a lot of traction for neural machine translation. In this setting, an 
encoder is used to map e.g. an English sentence into a vector. The decoder then conditions on this 
vector to generate a translation for the source English sentence. Several choices of encoder-decoder 
pairs have been explored, including ConvNet-RNN ifTOll . RNN-RNN ifTTl and LSTM-LSTM ifl^ . 
The source sentence representation can also dynamically change through the use of an attention 
mechanism ini to take into account only the relevant words for translation at any given time. In our 
model, we use an RNN encoder with GRU lfT4ll activations and an RNN decoder with a conditional 
GRU. This model combination is nearly identical to the RNN encoder-decoder of HD used in neural 
machine translation. GRU has been shown to perform as well as LSTM ID on sequence modelling 
tasks m while being conceptually simpler. GRU units have only 2 gates and do not require the use 
of a cell. While we use RNNs for our model, any encoder and decoder can be used so long as we 
can backpropagate through it. 

Assume we are given a sentence tuple (si_i, Si, s^+i). Let wj denote the f-th word for sentence Si 
and let x* denote its word embedding. We describe the model in three parts: the encoder, decoder 
and objective function. 


* A preliminary version of our model was developed in the context of a computer vision application |9l. 
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Query and nearest sentence 

he ran his hand inside his coat, double-checking that the unopened letter was still there . 
he slipped his hand between his coat and his shirt, where the folded copies lay in a brown envelope . 

im sure youll have a glamorous evening , she said , giving an exaggerated wink . 
im really glad you came to the party tonight, he said , turning to her . 

although she could tell he had n’t been too invested in any of their other chitchat, he seemed genuinely curious about this . 
although he had n’t been following her career with a microscope , he’d definitely taken notice of her appearances . 

an annoying buzz started to ring in my ears , becoming louder and louder as my vision began to swim . 
a weighty pressure landed on my lungs and my vision blurred at the edges , threatening my consciousness altogether . 

if he had a weapon , he could maybe take out their last imp , and then beat up errol and vanessa . 

if he could ram them from behind , send them sailing over the far side of the levee , he had a chance of stopping them . 

then , with a stroke of luck , they saw the pair head together towards the portaloos . 

then , from out back of the house , they heard a horse scream probably in answer to a pair of sharp spurs digging deep into its flanks . 

“ i ’ll take care of it, ” goodman said , taking the phonebook . 

“ i ’ll do that, ” julia said , coming in . 

he finished rolling up scrolls and , placing them to one side , began the more urgent task of finding ale and tankards . 
he righted the table , set the candle on a piece of broken plate , and reached for his flint, steel, and tinder . 


Table 2: In each example, the first sentence is a query while the second sentence is its nearest 
neighbour. Nearest neighbours were scored by cosine similarity from a random sample of 500,000 
sentences from our corpus. 

Encoder. Let wj,..., be the words in sentence Si where N is the number of words in the 
sentence. At each time step, the encoder produces a hidden state h* which can be interpreted as the 
representation of the sequence wj,... ,wj. The hidden state thus represents the full sentence. 
To encode a sentence, we iterate the following sequence of equations (dropping the subscript i): 


r* 

= cr(W^x*-f 

(1) 

z* 

= a(W,x‘+U,h‘-i) 

(2) 

h* 

= tanh(Wx*-fU(r*0h‘-i)) 

(3) 

h* 

= (l-z‘)0h‘"^-fz‘0h‘ 

(4) 


where h* is the proposed state update at time t, z* is the update gate, r* is the reset gate (0) denotes 
a component-wise product. Both update gates takes values between zero and one. 

Decoder. The decoder is a neural language model which conditions on the encoder output h^. The 
computation is similar to that of the encoder except we introduce matrices C^, C^. and C that are 
used to bias the update gate, reset gate and hidden state computation by the sentence vector. One 
decoder is used for the next sentence while a second decoder is used for the previous sentence 
Si-i- Separate parameters are used for each decoder with the exception of the vocabulary matrix V, 
which is the weight matrix connecting the decoder’s hidden state for computing a distribution over 
words. In what follows we describe the decoder for the next sentence Si+i although an analogous 
computation is used for the previous sentence Si_i. Let denote the hidden state of the decoder 
at time t. Decoding involves iterating through the following sequence of equations (dropping the 
subscript i + 1): 


r* 

= a(Wfx‘-i -f U^h*-i -f C^h,) 

(5) 

z* 

= CT(Wfx‘-i -f -f C^h,) 

(6) 

h* 

= tanh(WV-i -f U^(r‘ 0 h‘-i) -f Ch,) 

(7) 

h^i 

= (l-z‘)0h‘"^-fz‘0h‘ 

(8) 


Given the probability of word given the previous t — 1 words and the encoder vector is 

oc exp(v„^^h*+i) (9) 

where denotes the row of V corresponding to the word of An analogous computation 

is performed for the previous sentence Si_i. 

Objective. Given a tuple (si_i, Si, Si+i), the objective optimized is the sum of the log-probabilities 
for the forward and backward sentences conditioned on the encoder representation: 

^logP(w‘_^i|w4‘i,h,) -f ^logP(w‘_i|u;;i*i,h,) (10) 

t t 
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choreograph 

modulation 

vindicate 

neuronal 

screwy 

Mykonos 

Thpac 

choreography 

transimpedance 

vindicates 

synaptic 

wacky 

Glyfada 

2Pac 

choreographs 

harmonics 

exonerate 

neural 

nutty 

Santorini 

Cormega 

choreographing 

Modulation 

exculpate 

axonal 

iffy 

Dubrovnik 

Biggie 

rehearse 

##QAM 

absolve 

glial 

loopy 

Seminyak 

Gridlock’d 

choreographed 

amplitude 

undermine 

neuron 

zany 

Skiathos 

Nas 

Choreography 

upmixing 

invalidate 

apoptotic 

kooky 

Hersonissos 

Cent 

choreographer 

modulations 

refute 

endogenous 

dodgy 

Kefalonia 

Shakur 


Table 3: Nearest neighbours of words after vocabulary expansion. Each query is a word that does 
not appear in our 20,000 word training vocabulary. 


The total objective is the above summed over all such training tuples. 

2.2 Vocabulary expansion 

We now describe how to expand our encoder’s vocabulary to words it has not seen during training. 
Suppose we have a model that was trained to induce word representations, such as word2vec. Let 
Vu, 2 v denote the word embedding space of these word representations and let Vrnn denote the RNN 
word embedding space. We assume the vocabulary of Viu 2 v is much larger than that of Vmn- Our 
goal is to construct a mapping / : —>■ Vrnn parameterized by a matrix W such that v' = Wv 

for V € Vw 2 v and v' G Vmn- Inspired by ifTSll . which learned linear mappings between translation 
word spaces, we solve an un-regularized L2 linear regression loss for the matrix W. Thus, any 
word from Vw 2 v can now be mapped into Vmn for encoding sentences. Table Pshows examples of 
nearest neighbour words for queries that did not appear in our training vocabulm'y. 

We note that there are alternate strategies for solving the vocabulary problem. One alternative is to 
initialize the RNN embedding space to that of pre-trained word vectors. This would require a more 
sophisticated softmax for decoding, or clipping the vocabulary of the decoder as it would be too 
computationally expensive to naively decode with vocabularies of hundreds of thousands of words. 
An alternative strategy is to avoid words altogether and train at the character level. 


3 Experiments 

In our experiments, we evaluate the capability of our encoder as a generic feature extractor after 
training on the BookCorpus dataset. Our experimentation setup on each task is as follows: 

• Using the learned encoder as a feature extractor, extract skip-thought vectors for all sentences. 

• If the task involves computing scores between pairs of sentences, compute component-wise fea¬ 
tures between pairs. This is described in more detail specifically for each experiment. 

• Train a linear classifier on top of the extracted features, with no additional fine-tuning or back- 
propagation through the skip-thoughts model. 

We restrict ourselves to linear classifiers for two reasons. The first is to directly evaluate the rep¬ 
resentation quality of the computed vectors. It is possible that additional performance gains can be 
made throughout our experiments with non-linear models but this falls out of scope of our goal. Fur¬ 
thermore, it allows us to better analyze the strengths and weaknesses of the learned representations. 
The second reason is that reproducibility now becomes very straightforward. 

3.1 Details of training 

To induce skip-thought vectors, we train two separate models on our book corpus. One is a unidi¬ 
rectional encoder with 2400 dimensions, which we subsequently refer to as uni-skip. The other is 
a bidirectional model with forward and backward encoders of 1200 dimensions each. This model 
contains two encoders with different parameters: one encoder is given the sentence in correct order, 
while the other is given the sentence in reverse. The outputs are then concatenated to form a 2400 di¬ 
mensional vector. We refer to this model as bi-skip. For training, we initialize all recurrent matricies 
with orthogonal initialization M- Non-recurrent weights are initialized from a uniform distribution 
in [-0.1,0.1]. Mini-batches of size 128 are used and gradients are clipped if the norm of the parameter 
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Method 

r 

p 

MSE 






Method 

Ace 

FI 

Illinois-LH fTSl 

UNAL-NLP (m 

Meaning Factory 1201 

ecnuE] 

0.7993 

0.7538 

0.3692 

0.8070 

0.8268 

0.8414 

0.7489 

0.7721 

0.3550 

0.3224 

feats (2^ 

RAE-l-DP l24l 
RAE-l-feats i24l 
RAE-l-DP-l-feats 

73.2 
72.6 

74.2 
76.8 

83.6 

Mean vectors 1221 

DT-RNN (23) 

SDT-RNN |23l 

LSTM (22) 

Bidirectional LSTM 1221 
Dependency Tree-LSTM 1221 

0.7577 

0.6738 

0.4557 

0.7923 

0.7900 

0.8528 

0.8567 

0.8676 

0.7319 

0.7304 

0.7911 

0.7966 

0.8083 

0.3822 

0.3848 

0.2831 

0.2736 

0.2532 

FHS (25) 

PEl26l 

WDDP im 
MTMETRICS (HI 

75.0 

76.1 

75.6 

77.4 

82.7 

82.7 

83.0 

84.1 

uni-skip 

bi-skip 

73.0 

81.9 

uni-skip 

0.8477 

0.7780 

0.2872 

71.2 

81.2 

bi-skip 

combine-skip 

combine-skip-l-COCO 

0.8405 

0.8584 

0.8655 

0.7696 

0.7916 

0.7995 

0.2995 

0.2687 

0.2561 

combine-skip 
combine-skip -l- feats 

73.0 

75.8 

82.0 

83.0 





Table 4; Left: Test set results on the SICK semantic relatedness subtask. The evaluation metrics 
are Pearson’s r, Spearman’s p, and mean squared error. The first group of results are SemEval 2014 
submissions, while the second group are results reported by Hill . Right: Test set results on the 
Microsoft Paraphrase Corpus. The evaluation metrics are classification accuracy and FI score. Top: 
recursive autoencoder variants. Middle: the best published results on this dataset. 


vector exceeds 10. We used the Adam algorithm ini for optimization. Both models were trained 
for roughly two weeks. As an additional experiment, we also report experimental results using a 
combined model, consisting of the concatenation of the vectors from uni-skip and bi-skip, resulting 
in a 4800 dimensional vector. Since we are using linear classifiers for evaluation, we were curi¬ 
ous to what extent performance gains can be made by trivially increasing the vector dimensionality 
post-training of the skip-thought models. We refer to this model throughout as combine-skip. 

After our models are trained, we then employ vocabulary expansion to map word embeddings into 
the RNN encoder space. The publically available CBOW word vectors are used for this purpose 
The skip-thought models are trained with a vocabulary size of 20,000 words. After removing 
multiple word examples from the CBOW model, this results in a vocabulary size of 930,911 words. 
Thus even though our skip-thoughts model was trained with only 20,000 words, after vocabulary 
expansion we can now successfully encode 930,911 possible words. 

Since our goal is to evaluate skip-thoughts as a general feature extractor, we keep text pre-processing 
to a minimum. When encoding new sentences, no additional preprocessing is done other than basic 
tokenization. This is done to test the robustness of our vectors. 

3.2 Semantic relatedness 

Our first experiment is on the SemEval 2014 Task 1: semantic relatedness SICK dataset ll29l . Given 
two sentences, our goal is to produce a score of how semantically related these sentences are, based 
on human generated scores. Each score is the average of 10 different human annotators. Scores 
take values between 1 and 5. A score of 1 indicates that the sentence pair is not at all related, while 
a score of 5 indicates they are highly related. The dataset comes with a predefined split of 4500 
training pairs, 500 development pairs and 4927 testing pairs. All sentences are derived from existing 
image and video annotation datasets. The evaluation metrics are Pearson’s r, Spearman’s p, and 
mean squared error. 

Given the difficulty of this task, many existing systems employ a large amount of feature engineering 
and additional resources. Thus, we test how well our learned representations fair against heavily en¬ 
gineered pipelines. Recently, showed that learning representations with LSTM or Tree-LSTM 
for the task at hand is able to outperform these existing systems. We take this one step further 
and see how well our vectors learned from a completely different task are able to capture semantic 
relatedness when only a linear model is used on top to predict scores. 

To represent a sentence pair, we use two features. Given two skip-thought vectors u and v, we 
compute their component-wise product u ■ v and their absolute difference |m — u| and concatenate 

'"http:// code . google . com/p/word2vec/ 
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Sentence 1 

Sentence 2 

GT 

pred 

A little girl is looking at a woman in costume 

A young girl is looking at a woman in costume 

4.7 

4.5 

A little girl is looking at a woman in costume 

The little girl is looking at a man in costume 

3.8 

4.0 

A little girl is looking at a woman in costume 

A little girl in costume looks like a woman 

2.9 

3.5 

A sea turtle is hunting for fish 

A sea turtle is hunting for food 

4.5 

4.5 

A sea turtle is not hunting for fish 

A sea turtle is hunting for fish 

3.4 

3.8 

A man is driving a car 

The car is being driven by a man 

5 

4.9 

There is no man driving the car 

A man is driving a car 

3.6 

3.5 

A large duck is flying over a rocky stream 

A duck, which is large, is flying over a rocky stream 

4.8 

4.9 

A large duck is flying over a rocky stream 

A large stream is full of rocks, ducks and flies 

2.7 

3.1 

A person is performing acrobatics on a motorcycle 

A person is performing tricks on a motorcycle 

4.3 

4.4 

A person is performing tricks on a motorcycle 

The performer is tricking a person on a motorcycle 

2.6 

4.4 

Someone is pouring ingredients into a pot 

Someone is adding ingredients to a pot 

4.4 

4.0 

Nobody is pouring ingredients into a pot 

Someone is pouring ingredients into a pot 

3.5 

4.2 

Someone is pouring ingredients into a pot 

A man is removing vegetables from a pot 

2.4 

3.6 


Table 5: Example predictions from the SICK test set. GT is the ground truth relatedness, scored 
between 1 and 5. The last few results show examples where slight changes in sentence structure 
result in large changes in relatedness which our model was unable to score correctly. 


them together. These two features were also used by 


To predict a score, we use the same 


setup as EH. Let = [1,. .., 5] be an integer vector from 1 to 5. We compute a distribution p 
as a function of prediction scores y given hy pi = y — [yj if i = [yj + 1, pi = [yj — y + 1 if 
i = [yJ and 0 otherwise. These then become our targets for a logistic regression classifier. At test 
time, given new sentence pairs we first compute targets p and then compute the related score as p. 
As an additional comparison, we also explored a ppen ding features derived from an image-sentence 
embedding model trained on COCO (see section 3.4 1 . Given vectors u and v, we obtain vectors u' 
and v' from the learned linear embedding model and compute features u' ■ v' and \u' — v'\. These 
are then concatenated to the existing features. 


Table (left) presents our results. First, we observe that our models are able to outperform all 
previous systems from the SemEval 2014 competition. This is remarkable, given the simplicity 
of our approach and the lack of feature engineering. It highlights that skip-thought vectors learn 
representations that are well suited for semantic relatedness. Our results are comparable to LSTMs 
whose representations are trained from scratch on this task. Only the dependency tree-LSTM of ll22ll 
performs better than our results. We note that the dependency tree-LSTM relies on parsers whose 
training data is very expensive to collect and does not exist for all languages. We also observe 
using features learned from an image-sentence embedding model on COCO gives an additional 
performance boost, resulting in a model that performs on par with the dependency tree-LSTM. To 
get a feel for the model outputs, Table[^shows example cases of test set pairs. Our model is able to 
accurately predict relatedness on many challenging cases. On some examples, it fails to pick up on 
small distinctions that drastically change a sentence meaning, such as tricks on a motorcycle versus 
tricking a person on a motorcycle. 


3.3 Paraphrase detection 

The next task we consider is paraphrase detection on the Microsoft Research Paraphrase Cor¬ 
pus 1301. On this task, two sentences are given and one must predict whether or not they are 
paraphrases. The training set consists of 4076 sentence pairs (2753 which are positive) and the 
test set has 1725 pairs (1147 are positive). We compute a vector representing the pair of sentences 
in the same way as on the SICK dataset, using the component-wise product u ■ v and their absolute 
difference \u — v\ which are then concatenated together. We then train logistic regression on top to 
predict whether the sentences are paraphrases. Cross-validation is used for tuning the L2 penalty. 

As in the semantic relatedness task, paraphrase detection has largely been dominated by extensive 
feature engineering, or a combination of feature engineering with semantic spaces. We report exper¬ 
iments in two settings: one using the features as above and the other incorporating basic statistics 
between sentence pairs, the same features used by lED. These are referred to as feats in our results. 
We isolate the results and baselines used in ll24l as well as the top published results on this task. 

Table (right) presents our results, from which we can observe the following: (1) skip-thoughts 
alone outperform recursive nets with dynamic pooling when no hand-crafted features are used, (2) 
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coco Retrieval 


Model 

R@1 

Image Annotation 

R@5 R@10 

Med r 

R@1 

Image Search 

R@5 R@10 

Med r 

Random Ranking 

0.1 

0.6 

1.1 

631 

0.1 

0.5 

1.0 

500 

DVSAlin 

38.4 

69.6 

80.5 

1 

27.4 

60.2 

74.8 

3 

GMM+HGLMM ll32l 

39.4 

67.9 

80.9 

2 

25.1 

59.8 

76.6 

4 

m-RNN (33) 

41.0 

73.0 

83.5 

2 

29.0 

42.2 

77.0 

3 

uni-skip 

30.6 

64.5 

79.8 

3 

22.7 

56.4 

71.7 

4 

bi-skip 

32.7 

67.3 

79.6 

3 

24.2 

57.1 

73.2 

4 

combine-skip 

33.8 

67.7 

82.1 

3 

25.9 

60.0 

74.6 

4 


Table 6; COCO test-set results for image-sentence retrieval experiments. R@K is Recall@K (high 
is good). Med r is the median rank (low is good). 


when other features are used, recursive nets with dynamic pooling works better, and (3) when skip- 
thoughts are combined with basic pairwise statistics, it becomes competitive with the state-of-the-art 
which incorporate much more complicated features and hand-engineering. This is a promising result 
as many of the sentence pairs have very fine-grained details that signal if they are paraphrases. 


3.4 Image-sentence ranking 

We next consider the task of retrieving images and their sentence descriptions. For this experiment, 
we use the Microsoft COCO dataset [3j which is the largest publicly available dataset of images 
with high-quality sentence descriptions. Each image is annotated with 5 captions, each from dif¬ 
ferent annotators. Following previous work, we consider two tasks: image annotation and image 
search. For image annotation, an image is presented and sentences are ranked based on how well 
they describe the query image. The image search task is the reverse: given a caption, we retrieve 
images that are a good fit to the query. The training set comes with over 80,000 images each with 5 
captions. For development and testing we use the same splits as m- The development and test sets 
each contain 1000 images and 5000 captions. Evaluation is performed using Recall@K, namely the 
mean number of images for which the correct caption is ranked within the top-K retrieved results 
(and vice-versa for sentences). We also report the median rank of the closest ground truth result 
from the ranked list. 

The best performing results on image-sentence ranking have all used RNNs for encoding sentences, 
where the sentence representation is learned jointly. Recently, [32] showed that by using Eisher 
vectors for representing sentences, linear CCA can be applied to obtain performance that is as strong 
as using RNNs for this task. Thus the method of 0^ is a strong baseline to compare our sentence 
representations with. Eor our experiments, we represent images using 4096-dimensional OxfordNet 
features from their 19-layer model 051 . Eor sentences, we simply extract skip-thought vectors for 
each caption. The training objective we use is a pairwise ranking loss that has been previously 
used by many other methods. The only difference is the scores are computed using only linear 
transformations of image and sentence inputs. The loss is given by: 

^ ^ maxjO, a - s(Ux, Vy) -f s(Ux, Vy^,)} + X! X! “ s(Vy, Ux) -f s(Vy, Ux^)}, 

X fc y k 

where x is an image vector, y is the skip-thought vector for the groundtmth sentence, y^ are vectors 
for constrastive (incorrect) sentences and s(-,-) is the image-sentence score. Cosine similarity is 
used for scoring. The model parameters are {U, V} where U is the image embedding matrix and 
V is the sentence embedding matrix. In our experiments, we use a 1000 dimensional embedding, 
margin a = 0.2 and fc = 50 contrastive terms. We trained for 15 epochs and saved our model 
anytime the performance improved on the development set. 

Table [^illustrates our results on this task. Using skip-thought vectors for sentences, we get perfor¬ 
mance that is on par with both OTl and except for R@ 1 on image annotation, where other meth¬ 
ods perform much better. Our results indicate that skip-thought vectors are representative enough 
to capture image descriptions without having to learn their representations from scratch. Combined 
with the results of lf3^ . it also highlights that simple, scalable embedding techniques perform very 
well provided that high-quality image and sentence vectors are available. 
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(a) TREC (b) SUBJ (c) SICK 

Figure 2: t-SNE embeddings of skip-thought vectors on different datasets. Points are colored based 
on their labels (question type for TREC, subjectivity/objectivity for SUBJ). On the SICK dataset, 
each point represents a sentence pair and points are colored on a gradient based on their relatedness 
labels. Results best seen in electronic form. 


3.5 Classification benchmarks 


Eor our final quantitative experiments, we report results on several classification benchmarks which 
are commonly used for evaluating sentence representation learning methods. 

We use 5 datasets: movie review sentiment (MR) ll^ . customer product reviews (CR) OtI . sub¬ 
jectivity/objectivity classification (SUBJ) ll38l . opinion polarity (MPQA) and question-type 
classification (TREC) BOl . On all datasets, we simply extract skip-thought vectors and train a logis¬ 
tic regression classifier on top. 10-fold cross-validation is used for evaluation on the first 4 datasets, 
while TREC has a pre-defined train/test split. We tune the L2 penality using cross-validation (and 
thus use a nested cross-validation for the first 4 datasets). 


On these tasks, properly tuned bag-of- 
words models have been shown to per¬ 
form exceptionally well. In particular, 
the NB-SVM of BTl is a fast and ro¬ 
bust performer on these tasks. Skip- 
thought vectors potentially give an al¬ 
ternative to these baselines being just as 
fast and easy to use. Eor an additional 
comparison, we also see to what ef¬ 
fect augmenting skip-thoughts with bi¬ 
gram Naive Bayes (NB) features im¬ 
proves performance]^ 

Table presents our results. On most 
tasks, ^ip-thoughts performs about as 
well as the bag-of-words baselines but 
fails to improve over methods whose 
sentence representations are learned di¬ 
rectly for the task at hand. This indi¬ 
cates that for tasks like sentiment classi¬ 
fication, tuning the representations, even 
on small datasets, are likely to perform 
better than learning a generic unsupervised sentence vector on much bigger datasets. Einally, we 
observe that the skip-thoughts-NB combination is effective, particularly on MR. This results in a 
very strong new baseline for text classification: combine skip-thoughts with bag-of-words and train 
a linear model. 


Method 

MR 

CR 

SUBJ 

MPQA 

TREC 

NB-SVM pT) 

79.4 

81.8 

93.2 

86.3 


MNB (41| 

79.0 

80.0 

93.6 

86.3 


cBoW 

77.2 

79.9 

91.3 

86.4 

87.3 

GrConv 

76.3 

81.3 

89.5 

84.5 

88.4 

RNN 

77.2 

82.3 

93.7 

90.1 

90.2 

BRNf^ 

82.3 

82.6 

94.2 

90.3 

91.0 

CNN (41 

81.5 

85.0 

93.4 

89.6 

93.6 

AdaSent 

83.1 

86.3 

95.5 

93.3 

92.4 

Paragraph-vector (7] 

74.8 

78.1 

90.5 

74.2 

91.8 

uni-skip 

75.5 

79.3 

92.1 

86.9 

91.4 

bi-skip 

73.9 

77.9 

92.5 

83.3 

89.4 

combine-skip 

76.5 

80.1 

93.6 

87.1 

92.2 

combine-skip + NB 

80.4 

81.3 

93.6 

87.5 



Table 7: Classification accuracies on several standard bench¬ 
marks. Results are grouped as follows: (a): bag-of-words mod¬ 
els; (b): supervised compositional models; (c) Paragraph Vector 
(unsupervised learning of sentence representations); (d) ours. 
Best results overall are bold while best results outside of group 
(b) are underlined . 


3.6 Visualizing skip-thoughts and generating stories 

As a final experiment, we applied t-SNE ll4^ to skip-thought vectors extracted from TREC, SUBJ 
and SICK datasets and the visualizations are shown in Eigure Eor the SICK visualization, each 
point represents a sentence pair, computed using the concatenation of component-wise and absolute 
difference of features. Remarkably, sentence pairs that are similar to each other are embedded next 
to other similar pairs. Even without the use of relatedness labels, skip-thought vectors learn to 
accurately capture this property. 


^We use the code available at https : //github. cora/mesnilgr/nbsvm 
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Since our decoder is a neural language model, we can also generate from it. We can perform genera¬ 
tion by conditioning on a sentence, generating a new sentence, concatenating the generated example 
to the previous text and continuing. Since our model was trained on books, the generated samples 
reads like a novel, albeit a nonsensical one. Below is a 20 sentence sample generated by our model: 

she grabbed my hand . “ come on . ” she fluttered her bag in the air . “ i think we ’re at your place . i ca 
n’t come get you . ” he locked himself back up . “no . she will. ” kyrian shook his head. “ we met... that 
congratulations ... said no . ’’ the sweat on their fingertips’s deeper from what had done it all of his flesh hard 
did n’t fade . cassie tensed between her arms suddenly grasping him as her sudden her senses returned to its 
big form . her chin trembled softly as she felt something unreadable in her light. it was dark. my body shook 
as i lost what i knew and be betrayed and i realize just how it ended. it was n ’t as if i did n ’t open a vein . this 
was all my fault, damaged me . i should have told toby before i was screaming . i should’ve told someone that 
was an accident. never helped it. how can i do this, to steal my baby ’s prints ? ” 

4 Conclusion 

We evaluated the effectiveness of skip-thought vectors as an off-the-shelf sentence representation 
with linear classifiers across 8 tasks. Many of the methods we compare against were only evaluated 
on 1 task. The fact that skip-thought vectors perform well on all tasks considered highlight the 
robustness of our representations. 

We believe our model for learning skip-thought vectors only scratches the surface of possible objec¬ 
tives. Many variations have yet to be explored, including (a) deep encoders and decoders, (b) larger 
context windows, (c) encoding and decoding paragraphs, (d) other encoders, such as convnets. It is 
likely the case that more exploration of this space will result in even higher quality representations. 
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